fix: omit None fields from click inserts to stop bucket churn - #332

Problem

The clicks time-series collection averages 4.07 docs per bucket (66M docs across 16.2M buckets). The bucket index alone is 3.6GB, about half the database's total index footprint, and the resulting memory pressure is behind the recent Mongo incidents.

Root cause: click documents store empty analytics fields as explicit nulls (referrer: null, utm_source: null, bot_name: null). A time-series bucket cannot hold two BSON types in one column, so every alternation between a direct click (null referrer) and a referred click (string referrer) hard-closes the bucket with a schema-change rollover. Measured on prod: 1,035 of 10,254 inserts closed a bucket this way in one hour; the busiest link created 695 buckets in a single day from 1,106 null/string flips.

Fix

• ClickDoc.to_mongo() now serialises with exclude_none=True, so empty fields are absent instead of null. Scoped to clicks only; the shared MongoBaseModel.to_mongo() is unchanged.
• Bootstrap now creates the clicks collection with granularity: "hours", matching prod (collMod'd from "seconds" earlier). Fresh deploys previously got the churn-prone setting.

Why reads are safe

• Group-bys map null and missing identically via $ifNull sentinels (Direct / unknown / (none)).
• Sentinel filters already carry a {$exists: false} arm because clicks recorded before feat(analytics): add device and UTM click dimensions #259 lack the device/utm fields entirely; mixed shapes are already the norm in this collection.
• Every nullable ClickDoc field has a default, so from_mongo handles absent keys.

Verification

• Bucket behaviour on a local Percona Server for MongoDB 8.0: 200 alternating direct/referred clicks produce 200 buckets on main and 1 bucket with this change. Field-appears-later and int/double flips confirmed harmless; only null/type alternation closes buckets.
• Mixed-shape equivalence through the real StatsService over a collection holding 120 old-shape and 120 new-shape docs: 15 exact-number checks (totals, every dimension group-by, sentinel filters including referrer=Direct, non-sentinel filters, per-link path) all match independently computed values.
• Live app end to end, both sink modes: inline, and stream with the click worker consuming from Redis. Referred+utm, direct, and bot clicks via real HTTP; stored docs contain zero null values and only the keys with real values; public stats aggregate them correctly.
• Full suite: 3,639 passed, 5 skipped. Lint and format clean.

Existing buckets keep their shape (time-series buckets are immutable in place); this stops new churn, and a history repack can reclaim the index separately.

Summary by CodeRabbit

• Bug Fixes

  • Click data now omits optional fields when no value is provided, preventing unnecessary null values.
  • Empty metadata values are excluded from stored click records.
  • Click time-series storage now uses hourly granularity for improved alignment with production behavior.

• Tests

  • Added coverage verifying omitted and populated click fields are serialized correctly.
  • Updated click-handler tests to reflect the streamlined stored data format.